Using Delta Assertions to Detect Data Leakage

A long time ago, on a project far away, we were experimenting with different ways to clean up our test fixtures after our customer tests. Our tests were going against a database and were leaving objects behind. This caused us all sorts of problems with Unrepeatable Tests (see Erratic Test on page X) and Interacting Tests (see Erratic Test). We were also suffering from Slow Tests (page X).

We hit upon the idea of keeping track of all the objects we were creating in our tests by registering them with an Automated Teardown (page X) mechanism. Then we found a way to stub out the database with a Fake Database (see Fake Object on page X). Next, we made it possible to run the same test either against the fake database or the real one. This solved many of the interaction problems when running against the fake database but they still occured when running against the real database as we still had tests leaving objects behind and we wanted to know why. But first we had to know which tests were leaving objects behind.

The solution turned out to be pretty simple. In our Fake Database -- which was implemented using simple hash tables -- we added a method to count the total number of objects. We simply saved this value in an instance variable in the setUp method and used it as the expected value passed to an Equality Assertion (see Assertion Method on page X) called in the tearDown method to verify that we had cleaned up all the objects properly. (This is an example of using Delta Assertions (page X).) Once we had implemented this little trick, we quickly found out which tests were suffering from the Data Leak (see Erratic Test). We were then able focus our efforts on a much smaller number of tests.

To this day, we still find it useful to be able run the same test against the database and in-memory. And to this day, we still get the occasional test failing when the tearDown method inherited from our company-specific Testcase Superclass (page X) has a Delta Assertion failure.

The same idea could be applied to checking for memory leaks in programming languages with manual memory management (such as C++). Personally, I don't use such languages so I haven't verified this claim.

Page generated at Wed Feb 09 16:39:04 +1100 2011